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REMARKS 

Claims 1-23, 25-26, 29-31, 33-34, 37-41, and 45-49 are 
pending, with claims 1, 2, 12, 14, 18, 22, 24, 29, 32, 37, 40, 
41, and 42 being independent. Claims 24, 27, 28, 32, 35, 36 and 
42-44 were previously canceled without prejudice. Claims 1, 11, 
14, 40, 41, 48, and 49 have been cancelled by this amendment 
without prejudice. Claims 2, 5, 8, 10, 13, 15, -18, 20, 21, 23, 
29, and 37 have been amended. No new matter has been added. 
Reconsideration and allowance of the above -referenced 
application are respectfully requested. 

Information Disclosure Statement: 

Consideration of the previously submitted information 
disclosure statement is appreciated. As requested, a clean copy 
of citation AE is provided along with this response. Should 
there be any questions in view of this clean copy of the 
previously submitted reference, please contact the undersigned 
attorney at the earliest opportunity. 

Priority: 

The Office Action states that Applicant's claim for 
domestic priority is denied. Reconsideration of this is 
respectfully requested in view of the following. 
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First, it is noted that the lengths of the respective 
applications do not create a presumption of a lack of support 
for priority. Second, the Examiner's concern with respect to 
the length of the specification and the large number of 
references cited is understood. Nonetheless, these submissions 
were made in order to comply with the law. The undersigned 
certainly appreciates the undue burden placed on the Examiner 
and is willing to help as much as possible. However, there is 
no legal or administrative statutory basis to deny the priority 
claim based on "undue burden" . Should the Examiner need any 
further clarification or assistance after reviewing this 
response, please contact the undersigned attorney at the 
earliest opportunity . 

As requested, clarification of priority dates for the 
claims now pending is provided here. Claims 2-10, 12-13, 15-23, 
and 45-47 are entitled to the priority date of the original 
provisional application, U.S. Provisional Application Serial No. 
60/418,841, filed October 15, 2002 and entitled "AUGMENTED 
VIRTUAL ENVIRONMENTS (AVE) FOR VISUALIZATION AND FUSION OF 
DYNAMIC IMAGERY AND 3D MODELS". Claims 4, 8, and 2 0 are related 
to the details described in U.S. Pat. App. Serial No. 
10/278,349, filed October 22, 2002 and entitled "EXTENDABLE 
TRACKING BY LINE AUTO -CAL I BRAT I ON" . Nonetheless, claims 4, 8, 
and 20 are entitled to the priority date of the original 
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provisional application. (See the provisional application at 
page 3, col. 2, last paragraph, page 4, col. 1, first paragraph, 
and page 5, col. 2, first full paragraph.) 

Claims 25-26, 29-31, 33-34, 37-39 are not entitled to the 
priority of either U.S. Provisional Application Serial No. 
60/418,841, filed October 15, 2002 and entitled "AUGMENTED 
VIRTUAL ENVIRONMENTS (AVE) FOR VISUALIZATION AND FUSION OF 
DYNAMIC IMAGERY AND 3D MODELS" or U.S. Pat. App . Serial No. 
10/278,349, filed October 22, 2002 and entitled "EXTENDABLE 
TRACKING BY LINE AUTO- CALIBRATION" . 

Allowable Subject Matter: 

The indication that claims 12 and 22 have been allowed is 
appreciatively noted. Claims 29 and 37 would be allowable if 
rewritten to remove "at least" with respect to the number of 
recent image frames used. Claims 2 9 and 37 have been rewritten 
as suggested, and thus should now be in condition for allowance. 

Claim Rejections Under 35 U.S.C. 112: 

Claims 25-26, 29-31, 33-34, and 37-39 stand rejected under . 
35 U.S.C. 112, first paragraph, as allegedly failing to comply 
with the written description and enablement requirements. This 
contention is respectfully traversed. 
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In order to expedite prosecution, independent claims 2 9 and 
37 have been amended as suggested. Thus, each of claims 25-26, 
29-31, 33-34, and 37-39 should now be in condition for 
allowance . 

Claim Rejections Under 35 U.S.C. 103: 

Claims 1, 5, 11, 14 and 4 0 stand rejected under 35 U.S.C. 
103 (a) as allegedly being unpatentable over Kawasaki in view of 
Haala and Wang. Claims 2, 3, 7, 18, 41 and 45-49 stand rejected 
under 35 U.S.C. 103(a) as allegedly being unpatentable over 
Kawasaki, Haala and Wang, and further in view of Watanabe et al . 
Claim 4 stands rejected under 35 U.S.C. 103(a) as allegedly 
being unpatentable over Kawasaki, Haala and Wang, and Watanabe 
et al . , and further in view of Arpa et al . Claims 10, 15, 16 
and 17 stand rejected under 35 U.S.C. 103(a) as allegedly being 
unpatentable over Kawasaki, Haala and Wang, and further in view 
of Arpa et al . Claim 6 stands rejected under 35 U.S.C. 103(a) 
as allegedly being unpatentable over Kawasaki, Haala and Wang, 
and further in view of Weinhaus et al . Claim 19 stands rejected 
under 35 U.S.C. 103(a) as allegedly being unpatentable over 
Kawasaki, Haala and Wang, and further in view of Bar. Claim 8 
stands rejected under 35 U.S.C. 103(a) as allegedly being 
unpatentable over Kawasaki, Haala and Wang, and further in view 
of Jiang et al . Claim 13 and 23 stand rejected under 35 U.S.C. 
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103(a) as allegedly being unpatentable over Kawasaki, Haala and 
Wang, and further in view of Pryor et al . Claims 45, 4 7 and 49 
stand rejected under 35 U.S.C. 103(a) as allegedly being 
unpatentable over Kawasaki, Haala Wang, and Watanabe, and 
further in view of Chen et al . These contentions are 
respectfully traversed. 

The rejections of claims 1, 11, 14, 40, 41, 48, and 49 have 
been obviated by the cancellation of these claims. Claims 2-10, 
13, 15-21, 23, and 45-47 now depend from an allowable base claim 
and should be allowable for at least this reason. 

It is believed that all of the pending claims have been 
addressed. However, the absence of a reply to a specific issue 
or comment does not signify agreement with or concession of that 
issue or comment. Because the arguments made above may not be 
exhaustive, there may be reasons for patentability of any or all 
pending claims (or other claims) that have not been expressed. 
Finally, nothing in this paper should be construed as an intent 
to concede any issue with regard to any claim, except as 
specifically stated in this paper, and the amendment of any 
claim does not necessarily signify concession of unpatentability 
of the claim prior to its amendment. 

It is respectfully suggested for all of these, reasons, that 
the current rejections are overcome, that none of the cited art 
teaches or suggests the features which are claimed, and 
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therefore that all of these claims should be in condition for 
allowance. A formal notice of allowance is thus respectfully- 
requested. 

Please apply any necessary charges or credits to Deposit 
Account No. 06-1050. 
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Abstract 

A major problem challenging opera designers is the inability to co- 
ordinate lighting, projection sysiems, and set designs in the prelim- 
inary planning phase. New computer graphics techniques/ which 
provide the set and lighting designer the opportunity to evaluate, 
test, and control opera designs prior to the construction of full scale 
systems are presented. These techniques — light source input, simu- 
lation of directional lighting, modeling of scenic projection systems, 
and full three-dimensional simulation — show the potential for the 
use of computer graphics in theater design. 

The light source input component consists of a program for as- 
signing light source attributes with a set of theater lighting icons. 
This module allows a designer to specify light source characteristics 
in a way familiar to the discipline and to make preliminary evalua- 
tions of the lighting conditions. 

An extended progressive radiosity method is introduced to sim- 
ulate the directional lighting characteristics which are specified by 
the input program. 

A new projection approach is presented to simulate the optical 
effects of scenic projectors. In addition, a solution to the distortion 
problem produced by angular projections is described. 

The above components are integrated to produce full three- 
dimensional simulations of the global illumination effects in an 
opera scene. 

CR Categories and Subject Descriptors: 1.3.0 (Computer 
Graphics]: General; 1.3.7 [Computer Graphics): Three Dimen- 
sional Graphics and Realism; J. 2 [Computer Applications]: Per- 
forming Arts. 

General Terms: Algorithms 

Additional Keywords and Phrases: opera and stage design, angu- 
lar projection, simulation, radiosity, directional light sources, tex- 
ture mapping. 

1 Introduction 

Opera stage design is an extremely difficult task as, in addition to 
the standard architectural and aesthetic considerations, a number of 
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additional issues are present, such as dynamic and intricate lighting 
and sets, projected background scenery, changing focus of attention, 
manipulation of implied perspective, multiple viewing points, mo- 
tion of performers, and synchronization with music. Stage and light- 
ing designers, as well as conductors, rarely have the opportunity to 
evaluate these effects together. Consequently, stage set and lighting 
designs are currently developed separately — being combined only 
in the last step of the process. 

Presently, the only feasible method available for combining a 
limited stage and lighting design is the construction of small scale 
models. While this process does give some insight into the visual 
impact of the final production, it is a laborious, costly, incomplete, 
and time-consuming endeavor. Furthermore, because of their small 
scale, these models are so inadequate for the evaluation of complex 
lighting effects that they are not commonly used. Thus, in practice, 
the stage and lighting designers will often work in isolation from 
each other. The bulk of the lighting designer's task, then, occurs 
at the last minute — after the sets are assembled and in place on the 
stage. 

The primary objective of this paper is to provide the stage and 
lighting designer the opportunity to design and evaluate the lighting 
and projected scenery prior to the actual implementation. In partic- 
ular, techniques for light source descriptions and specification, the 
simulation of directional lighting, and the modeling of scenic pro- 
jection systems have been developed. In addition, a solution for the 
distortion problem in angular projections is introduced. The proce- 
dures have been combined to provide full three-dimensional simula- 
tions so that the proposed design strategy can be evaluated from any 
viewer position. The variables are the positions of the stage sets, the 
locations, orientation, spatial emittance, and color of the lights, the 
number of lights which are illuminated, the background projection 
sysiems and scenery, all within the given theater geometry. 

Three famous opera houses have been selected to demonstrate 
the system: the Metropolitan Opera at Lincoln Center in New York 
City, La Scala in Milan, Italy, and the Staatsoper in Vienna, Austria. 
Due to space limitations, only the Metropolitan Opera is illustrated. 

2 Input for Lights 

In general, light sources have well-defined finite geometries that 
greatly affect the distribution of the light emitted from the source. 
There are three types of abstract emissive geometries: point sources 
(zero dimensional), linear sources (one dimensional), and area 
sources (two dimensional) |23|. The light sources used in opera 
production can be treated as point sources, since the lights are very 
small and are located at a significant distance from the stage. 

Perhaps the most important characteristic of a luminaire that must 
be included in a complete model of a light source is the luminous 
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intensity distribution. In contrast to the assumption typically used 
in computer graphics, most of the lights used in opera production 
do not emit light of constant intensity in all directions. 

A non-uniform intensity distribution must be specified* which 
describes the variations of the emitted intensity with direction. 
The lighting industry uses goniometric diagrams to represent these 
vector-valued functions for easy interpretation [3]. These diagrams 
represent a planar slice through the vector field and thus plot the 
relative intensity as a function of angular direction (Figure 1 ). For 




Figure 1: A sample emission distribution, (a) Polar goniometric 
diagram, (b) Corresponding cartesian diagram. 



luminaires with concentrated beams, such as spotlights, cartesian 
coordinates are preferred because of the need for more precision 
than a polar curve allows. 

2.1 Instruments and Lamps 

In lighting design foT opera, many different types of luminaires are 
used. Although there appear to be a large number of different in- 
strument styles used, each style is a variation on five particular in- 
struments: the ellipsoidal reflector spotlight; the Fresnel spotlight; 
the striplight; the ellipsoidal reflector floodlight; and the beam pro- 
jector. The optical characteristics of each of these instruments (and 
variations thereof) have been modeled. 

22 Assigning Light Source Attributes 

An interactive graphical program has been developed to allow one 
to design a lighting scenario. While a final lighting layout is primar- 
ily a tool for communicating the designer's concept and intentions 
to the electricians, lighting crews, and board operators, who must 
"hang" the design and execute it in a performance, this program 
provides the means to develop ideas, experiment, move and change 
instruments and their attributes, and iteratively refine a design. 

The input program allows for the complete specification of at- 
tributes for stage lighting. The user can specify the position of each 
instrument, its intensity pattern, color, projection pattern, and the 
area which it illuminates in addition to indicating its height above 
the floor and the angles of the beams of light. As the parameters as- 
sociated with a light are adjusted, the lamp is instantly updated with 
the resulting beam and field angles as well as throw distance. This 
feature makes it possible to combine light sources and evaluate the 
design implications (e.g. if their beams overlap). Thus, while still 
in the modeling phase, one has a good idea of the overall lighting 
scheme. Once a preliminary design is specified, the user can simu- 
late the illumination effects and, with a separate program, view the 
results to further refine the lighting design. 

An attempt has been made to carefully design the graphical in- 
terface so the process of assigning attributes is similar to the way in 
which it is physically performed. A menu is available which con- 
tains a two dimensional graphical representation of the five major 
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categories of lights used in opera production. Once a category has 
been selected, it is possible to choose from a variety of instrument 
types and manufacturers within that category. 

Most of the luminaires used in theater production have a spot 
and flood focus intensity distribution associated with them. When 
a lamp has been selected, a two dimensional icon is drawn in one 
window, and the components of the lamp which move during the 
focusing of that particular instrument can be varied interactively, 
making it possible to focus the instrument to the desired setting 
(Figure 2). To specify the intensity distribution, one window shows 
either a cartesian or polar goniometric diagram of the current lu- 
minous intensity distribution or candle power distribution curve for 
the light source. This diagram is updated continuously as the instru- 
ment is focused. To scale the distribution, the maximum intensity 
which the lamp emits at the center of its beam is specified in units 
of candelas. 

Each light source can have a unique partem or slide. A library of 
patterns and slides to be used in projection has been compiled. The 
designer can select a pattern from the library and associate it with a 
given light source. 

Color can be controlled by placing a transparent color filter be- 
tween the light source and the receiving surface(s). Using the filter 
section of the input program, an interactive color tool allows the 
user to vary the characteristics of the filter used to color the light 
emitted by a lamp. 

It is possible to position a lamp at any location relative to the stage 
environment. Most light sources are positioned on the light bridges, 
but occasionally they are placed on the front edge of the stage as 
footlights or on temporary ladder-like structures along the sides. To 
aid the user in positioning the lamps, one viewport displays the light 
source with three dimensional transparent cones attached to it (op- 
tionally displayed) in the model of the stage. These cones represent 
the beam and field angles as well as the throw distance of the in- 
strument (see Appendix A). As the lights are positioned relative to 
the stage area, the cones allow the user to visualize the direction in 
which the light will be emitted as well as how much illumination a 
particular area will receive. 



3 Simulation of Theater Lighting Conditions 

Radiosity methods, derived from the field of radiative heat transfer, 
have been successfully applied to the area of realistic image synthe- 
sis [4, 5, 6, 9, 16, 24]. The radiosity method has the attractive char- 
acteristic of providing a view-independent solution. Hence, once 
the solution has been performed, a hardware tenderer can be used 
to display the scene from changing viewpoints at interactive rates. 

3.1 Modeling Directional Light Sources with Progres- 
sive Radiosity 

The progressive radiosity method [4] can be extended to account for 
directional variations in a light source with non-uniform emission 
distributions (Figure 1). In this implementation, the form-factors 
are computed using the ray-traced form-factor approach proposed 
by Wallace et al.[24]. To account for the variation in light source di- 
rectionality, the form factor from the light source to a vertex is com- 
puted as usual and then weighted by a directionality scaling factor, 
s. Each directional light source has a distribution associated with it 
which describes its normalized light source intensity versus angle. 
The value of s is obtained from this distribution for each element 
vertex based on the direction 0 2 (the angle between the direction 
vector of the light source and the direction of an element vertex). 

In this way, the amount of light which is transferred from the light 
source to the environment is weighted according to the directional 
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(a) 



Figure 2: Light Source Input and Attribute Assignment, (a) The 
flood foci. 




Figure 3: Modeling directional light sources with progressive ra- 
diosity. 



distribution (Figure 3). The illumination received at vertex 1 from 
a spot light 2 can be represented by a weighted form-factor based 
on that light source's emission distribution: 




(b) 



spotlight - spot and flood foci, (b) The beam projector - spot and 
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For directional light sources, a single ray is traced from each ver- 
tex to the light for shadow testing. A single ray is sufficient because 
point sources are used. The amount of energy per unit area received 
at each vertex is then weighted as shown above. Figure 4 shows a 
radiosity rendering of several directional light sources. 

In this method, the directional light sources initially shoot out 
their directional energy once, and calculations for any subsequent 
radiant energy exchange due to secondary reflections can be treated 
in the standard manner. 

4 Projected Scenery 
4.1 Overview 

An effective method of creating a scenic background is to project a 
slide onto a neutral surface. Architectural features, general views, 
natural objects, cloud formations, and similar objects can be pro- 
jected on to a backdrop. Projected scenery is fundamentally dif- 
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Figure 4: Radiosity rendering of directional light sources. 



ferent from built or painted scenery in that it achieves its effects 
through the use of light. Because color in light is more brilliant than 
in paint, and has an unlimited value scale by comparison, its use in 
projection is far more dramatic and eye-catching. Additionally, it 
can give a greater illusion of size and depth in a setting [27]. In a 
sense, projection techniques create an imaginary space which ex- 
tends the bounds of the real space denned by the set geometry. Fur- 
thermore, they permit not only rapid changes, but also gradual tran- 
sitions from one setting to the next, such as from daylight, through 
sunset to night — all with the same background [II]. 

Typically, the picture to be projected is painted by hand or photo- 
graphically produced. Slides for scenic projectors range in size from 
5 to 10 inches square. Reduced to its simplest elements, the projec- 
tion process consists of a light source, the object or slide, the pro- 
jected image, and the projection surface [18]. There are two types 
of projection: shadow projection and tens projection. The term lens 
projection is used to define all projections obtained using one or 
more lenses; shadow projection is used to describe projections ob- 
tained without the use of lenses [26]. 

4.2 Simulation of Projected Scenery 

The simulation of background scenery using slide projection tech- 
niques is common in opera production, but is new to the held of 
computer graphics. It is important to derive methods for projec- 
tion simulation which will maintain the resolution and quality of 
the original slide, provide soft-focusing according to the optics of 
the projection system, allow arbitrary geometries for the receiving 
surfaces, mimic the correct dispersion and attenuation of light, and 
be computable in tractable amounts of time. 

4 J A Radiosity Projection Technique 

This section describes an extension to the progressive radiosity al- 
gorithm which allows for the projection of scenery. The energy re- 
ceived at the surface of the backdrop from the projection is a func- 
tion of the emission distribution of the light source, the transmissi v- 
ity values of the slide, and the orientation and distance of element 
vertices on the backdrop relative to the projection system. The tech- 
nique (Figure 5) can be expressed as follows: 

1. A two-dimensional array, or "texture map," of values is ob- 
tained by scanning photographs or artistic renderings of actual 
images to be projected. 

2. The backdrop/receiving surface is discretized into a series of 
element vertices, the locations of which are determined based 





projector 
(poinl light source) slide" 



Figure 5: Radiosity Scenic Projection Technique. The backdrop 
is discretized according to the projected resolution of the slide. The 
initial radiosity at each element vertex is then based on the emission 
of the light source, the transmissivity of a point in the slide, and the 
color of the backdrop. 



on the intersection of casting a single ray from the projector 
through each point of the slide to the backdrop. The resolution 
of the slide varies according to the desired fuzziness or clarity 
of the projected image on the backdrop (typically a slide with 
a resolution between 50x50 and 100x100 = 2,500 to 10,000 
element vertices is used). 

3. The radiosity of each of these vertices is then based on the 
color/transmissivity of the relevant point in the slide, the back- 
drop color, and the emission of the light source in the direction 
of the element vertex. 

The initial radiosity at a vertex on the screen due to the projection 
(source 2) only is expressed by this modified radiosity equation: 



COS0| COS02 /Q , . 

'P\E 2 ^ s(0i ) t q 



7rr- 



(2) 



where, 

£2 = energy emitted by the point light source, 
t q = transmissivity at a point q in the slide. 

It should be noted that the effect of a directional light source, 
3(62) (projector) is also included in the above equation. 

It is also possible, using computer graphics, to model the depth 
of focus of the projection system. One can "blur" the slide using a 
filter, sized according to the distance of the projector to the back- 
drop, so that the entire projection has the same out-of-focus effect. 
Figure 6 shows a sample projection which was generated using this 
method. 

The radiosity projection technique closely simulates the real pro- 
jection system in that it yields a view-independent solution with in- 
terpolation at the backdrop based on projection characteristics at the 
maximum stored resolution of the slide. In addition, the backdrop 
is subdivided according to the projection itself, rather than accord- 
ing to a separate and unrelated element meshing. Finally, the light 
attenuation of the projector is modeled precisely. 

4.4 A General Solution for the Distortion Problem in 
Angular Projection 

It is usual to place the projector(s) behind the proscenium, hidden 
from the view of the audience, and to project downward at an an- 
gle to the backdrop. Furthermore, it is common for the backdrop to 
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Figure 6: Radios ity Scenic Projection 




Figure 7: Uncorrected Angular Projection - A square slide in the 
projector will produce a distorted backdrop image for a viewer. 



be non- planar and shaped in some way (e.g. a curved cyclorama). 
A slide in the projector will therefore produce a distorted backdrop 
image, which increases in size with the distance from the projector 
to the backdrop as shown in Figure 7. The problem then is how to 
predistort a slide such that, when projected and viewed from the po- 
sition of an "ideal viewer," the backdrop image appears undistorted. 

Currently, when a scene is to be projected, the lighting designer 
waits until the sets are built and in place. Then, a slide (known as a 
raster) of a regular grid of guide marks is projected from the posi- 
tion where the projectors) will be located. Next, the resulting pro- 
jected image (now a distorted grid) is photographed and examined 
from various viewer positions. Based on the distortion apparent in 
the projected grid at an ideal viewer location, a predistorted slide is 
produced using the distorted grid lines as guides. When the slide is 
projected, the viewer is presented with an undistorted scene. Gen* 
erally, several iterations of this process are necessary to achieve the 
desired slide. This trial and error process is time-consuming, labor 
intensive, and restricted to a given set/projection geometry. 

This process can be computer simulated using a digital image 
warping technique— providing a designer the opportunity to pre- 



view a projection and produce the distorted slide long before sets are 
built. The algorithm involves modeling the distortion which would 
result based on the geometry of a hypothetical projection schema 
and generating a pre-distorted slide to counteract the distortion. 

Image* warping is a growing branch of image processing thai 
deals with the geometric transformation of digital images) 28 1. A ge- 
ometric transformation is an operation that redefines the spatial rela- 
tionship between points in an image. The basis of geometric trans- 
formations is the mapping of one coordinate system onto another 
This mapping is defined by means of a spatial transformation — a 
mapping function that establishes a spatial correspondence between 
all points in (he input and output images. 

Geometric transformations were originally introduced to correct 
the distortions introduced by remote sensing methods [2, 10|. This 
process involved estimating the distortion model, usually by means 
of a set of reference points. The algorithm presented below differs 
from these previous techniques in three important respects. First, 
in this method, one begins with the undistorted image and models 
the distortion which is introduced. Second, the method is very gen- 
eral in that it does not rely on a particular sensor: it can be used to 
model any projection system and can account for a projection from 
any location. Third, this technique involves a two-step approach, 
because the new value at each point is not merely taken from the 
mapped coordinates (as is usually the case) but is instead derived 
from a second mapping (ideal/desired transformation). 

4.4.1 Definitions 

In discussing the problem of an angular projection, it is useful to 
introduce definitions of the three pertinent coordinate systems. A 
point on the slide in the projector is referred to as x P y f ,z p . a point 
in the projected image on the backdrop as XbybZb, and a point in the 
viewer's/spectator's space as x,.y t z r . The physical slide defines the 
continuous function of ix p y p z v ) specifying a color at each position. 

The ideal situation would be one in which the projection system 
would associate points in the projector (slide) and viewer space such 
that the viewer simply sees a window into the original picture. This 
operation would define an afflne transformation T such that, 

(x v y,z,) = (x p y p z p )T (3) 

The color (intensity) at some point x t y,z, would then simply de- 
pend on the color of the slide at point (x P y r z P ) - (x,.y t z,.)T~ l . 

color ix r yrZ,.) = F\orightol-slifir((x,.y t z t .}T~ 1 )] (4) 

where the function F describes the effect of light attenuation and 
illumination (Its exact form does not need to be specified here). 

However, in reality, a point on the slide x p y p z v will be first 
mapped to the backdrop by the projection, according to some pro- 
jection transformation, P. The resulting point on the backdrop will 
in turn be transformed to the viewer's space by the usual perspective 
viewing transformation. V. Thus, 

(x b yhZb) = (x p y p z p )P (5) 

and. 

(x v y v z v ) = (XbybZb)V = ix P y P z v )PV = (x p y p z p )D (6) 

The transformation, D- PV, represents the combined mapping 
from the projection and viewing systems or the distortion, which is 
ultimately seen by the viewer. There is a parallel between the real 
distortion. D and the ideal transformation. T. 

The problem is to create a pre-distorted slide such that, after ap- 
plying the physical transformation D to this slide, the color at each 
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point in the viewer space is the same as the color obtained by ideally 
transforming the original picture using the transformation T: 

V(x v j/„2 V ) F{original_slide((x v y v z v )T~ 1 )) = 

F(distorted.slide{(x v y v z v )D~ x )) (7) 

A simple way of guaranteeing this is to define the distorted slide as: 



V(ipVpip) distortedjslide{x p y p z p ) = 

original Midc{(x p y p z p )DT " 1 ) (8) 

4.4.2 Producing a Pre -distorted Slide 

An algorithm to produce a pre-distorted slide which simulates the 
actual process used by lighting engineers is presented below. By us- 
ing Equation (8), the color at every point on the slide is computed. 
In order to evaluate this equation, a way to compute the distortion 
D is needed. In general, D is a complex non-linear transformation, 
and thus it is not practical to evaluate it analytically. The algorithm 
proposed computes D exactly for a small number of points and uses 
a linear interpolation to approximate D at all other points. A regular 
orthogonal grid (the resolution of which can be varied) can be used 
as a means to quantify the distortion function, D. Rays are sent out 
from the projector at regular intervals (based on the resolution of the 
grid and the beam angle of the projector) to the projection surface. 
By transforming the set of grid points from the projector coordi- 
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Figure 8: Modeling the distortion in a projection. A regular grid 
is projected from the projector to the backdrop and mapped to the 
viewer. The projection system P maps a point from the projector 
to the screen. The observer's eye-system V maps a point from the 
screen to the viewer. 
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Figure 9: Process for rinding the color of a point in the pre-distorted 
slide. 



1. Find which grid cell of the undistorted grid the point is in, 

2. Find the coordinates (u, v) of the point within the undistorted 
grid cell (the local coordinates within the cell), 

3. Compute the coordinates of the transformed point (x v y v z v ) in 
the distorted grid. This computation is accomplished by bilin- 
ear interpolation (or a higher order interpolation if necessary) 
using the four transformed corners of the cell. (At this point 
the effect of D has been evaluated), 

4. Find the point (x p y' p z p p ) in the original image associated to 
x v y v z v by the ideal transformation T~ x . This is a standard 
windowing operation involving a simple affine transforma- 
tion, 

5. The color of the distorted slide at point (x p y p z p ) is the color 
of the original slide at point (x p y p z p ). 

The result of this procedure is the predistorted slide. 

Figure 10 shows a projected image of a New York City sky- 
line onto a curved backdrop along with an illustration of the 
corrected/pre-distorted slide which was generated using the above 
method. 



5 Simulation Results 



nates, x p y p z p to the coordinates on the backdrop/projection surface 
xbybZb the function P is simulated. Next, the projected points are 
transformed into the viewer's coordinates, x v y v z v , based on a cam- 
era specification which represents the view of an ideal spectator. 
This transformation represents the function V. 

The relationship between the original undistorted grid (the set 
of points x p y p z p ) and the resulting points as seen by the viewer 
(x v y v z v ) represents the distortion, D, which was produced by the 
projection (Figure 8). 

To compute the final slide, the following steps are taken: 
Fust, the virtual, undistorted grid is overlayed onto the slide to be 
generated as shown in Figure 9. Next, for each point of the slide. 



The geometric information in the opera hall is subdivided into two 
parts, the geometry of the permanent structure and the geometry and 
position of the stage sets. 

Fixed Geometry - Definition of the Opera Hall 

The geometry of the structure, such as the shell or roof over the 
stage, as well as the auditorium in general, is fixed. Since the 
focus of the simulation is only on the stage, a detailed model of 
the stage portion of the hall has been constructed along with an 
abstract overall building model. This detailed model includes the 
proscenium and stage with operable stage lifts as well as the light 
bridges/gantries where the lights are positioned. The model was 
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Figure 10: Radiosity scenic projection of a New York City skyline using a predistorted slide. The projection is from a height of 5 1.5 feet 
down at an angle to the curved backdrop, (a) Abstract plan of the environment showing the location of the projector, an ideal viewer, and the 
shape of the backdrop, (b) Final projected image on the backdrop, (c) Predistorted slide. 



made to scale using precise dimensions from drawings and pho- 
tographs received from the Metropolitan Opera House in New York 
City. 

Variable Geometry - Definition of the Stage Sets 

Detailed models of the stage sets constitute the second part of the 
geometric definition. Most of the sets have been defined as extruded 
contours. Once a set has been modeled, it is possible to position it 
at any location on the stage and to view it from any position in the 
hall. 



signer a unique opportunity to design, preview, and assess an opera 
design prior to the construction of full-scale systems. The results of 
this research clearly demonstrate that Che use of computer graphics 
in theater design holds great promise, particularly since these tech- 
niques afford the opportunity for aesthetic evaluations to be made 
early in the design process and consequently allow many design pro- 
fessionals to work in unison in the preliminary design phase. 

Future directions include the use of a higher order interpolation 
scheme for the projection distortion algorithm, anti-aliasing in the 
generation of a pre -distorted slide, and the control of the lighting as 
a function of time. 



Examples of Lit Opera Scenes 

To demonstrate the usefulness of the techniques, three full simu- 
lations of real productions from the Metropolitan Opera have been 
generated and are shown in Figures 11,12 and 13. Each of these ex- 
amples shows the specialized lighting effects— projected scenery, 
angular projection solutions using pre-distorted slides, and stage 
lighting— combined with high quality view-independent Simula- 
tions for practical use in opera design. 

Figure 1 1 shows two views for GUnther Schnetder-Siemssen's 
design for the Palazzo on the Grand Canal in Venice from Les Con- 
tes d Hoffmann. The left and right sides of the set are modeled and 
texture-mapped. The facades in the center background were pro- 
jected from stage left at a height of 46.5 feet onto a curved back- 
drop using a pre-distorted slide. Striplights are used to provide an 
overall wash of light on the set. A series of floodlights were used 
to simulate moonlight on the facades to the right. A high intensity 
spotlight produces the effect of the lantern in the doorway. 

Figures 12 and 13 show two distinct lighting schemes for Franco 
Zeffirelli's design of a Parisian garret from La Bohime. A promi- 
nent feature in these simulations is the sky, which was projected 
using two angular projections. The projectors are positioned on a 
lighting bridge at a height of 53 feet on the left and right sides of 
the stage. The slides were predistorted to account for the angular 
projections. A variety of stage lights are positioned on the bridges 
and used to illuminate the set. 



6 Conclusions 

A set of computer graphics techniques for the design and simulation 
of opera lighting effects has been presented. The light input, pro- 
jection, and simulation components give the stage and lighting de- 



7 Acknowledgements 

The authors are extremely grateful to the Hewlett Packard Corpora- 
tion and the Digital Equipment Corporation for donating the equip- 
ment which was used for the simulations and displays. The authors 
also thank the National Science Foundation for long term support 
of the research at the Program of Computer Graphics. The Robert 
James Eidlitz Traveling Fellowship provided funding for a site visit 
to the Staatsoper, La Scala. and the Metropolitan Opera. Special 
thanks to Gil Wechsler and Wayne Chouinard at the Metropolitan 
Opera for providing documentation of real set and lighting designs 
and for their helpful discussions and input into this research. Mark 
Shepard modeled the La Bohime environment used in the simula- 
tions. Thanks to Kurk Dorse y, Hurf Sheldon, Ben Trumbore, Kevin 
Novins, Harold Zatz, Ellen French, and Fran Brown for their gen- 
eral assistance. All images were photographed by Emil Ghinger. 



References 

[ 1 ] Baum, Daniel R., Holly E. Rushmeier, and James M. Winget. 
"Improving Radiosity Solutions Through the Use of An- 
alytically Determined Form-Factors," Proceedings of SIG- 
GRAPH'89 (Boston, Massachusetts, July 31 - August 4, 
1989), in Computer Graphics, 23(3), July 1989. pages 325- 
334. 

[2] Bernstein, Ralph. Digital Image Processing of Earth Obser- 
vation Sensor Data, IBM J. Res. Develop., 1976. 

[3] Cay less, M. A. and A. M. Marsden. Lamps and Lighting. Ed- 
ward Arnold, London. 1983. 



BEST AVAILABLE COPY 



47 




Figure 11: Two views of the Palazzo on the Grand Canal in Venice from Les Contes d' Hoffmann. The left and right sides of the set are 
modeled. The canal scenery in the center background was projected at an angle onto a curved backdrop using a predistorted slide. A series of 
floodlights are used to simulate a dim moonlight on the facades to the right. A high intensity spot is used to light the entrance to the left. 



48 



@ ^ Comouter Graohics. Volume 25. Number 4. Julv 1 991 





Figure 12 (top and left): Two views of a garret above the rooftops of 
Paris from La Bohime (evening). The scene contains two angular pro- 
jections (using predi stoned slides) for the sky and a variety of spots and 
floods. The focus of the lighting is on the entries and the interior of the 
garret. Figure 13 (right): Alternative lighting scheme for the garret from 
La Boheme (daytime). The projected slides have been re-colored to ob- 
tain the day effect, and additional colored spot lighting emphasizes the 
center of the set and the right entry. 



BEST AVAILABLE COPY 



□ 



- SIGGRAPH '91 Las Vegas, 28 July-2 August 1991 



[4] Cohen, Michael R, Shenchang Eric Chen, John R. Wallace, 
and Donald P. Greenberg. "A Progressive Refinement Ap- 
proach to Fast Radiosity Image Generation," Proceedings 
of SIGGRAPH'88 (Atlanta, Georgia, August 1-5, 1988), in 
Computer Graphics, 22(4), August 1988, pages 75-84. 

[5] Cohen, Michael F. and Donald P. Greenberg. **A Radiosity 
Solution for Complex Environments," Proceedings of SIG- 
GRAPH*85 (San Francisco, California, July 22-26, 1985), in 
Computer Graphics, 19(3), July 1985, pages 3 1-40. 

[6] Cohen, Michael F., Donald P. Greenberg, David S. Immel, and 
Philip J. Brock. "An Efficient Radiosity Approach for Realis- 
tic Image Synthesis,'* IEEE Computer Graphics and Applica- 
tions, 6(3), March 1986, pages 26-35. 

[71 Dorsey, Julie O'B. Computer Graphics for the Design and 
Visualization of Opera Lighting Effects, Master's thesis, Pro- 
gram of Computer Graphics, Cornell University, Ithaca, NY 
14853, January 1990. 

[8] Giiette, Michael J. Designing With Light, Mayfield Publishing 
Company, Palo Alto, 1978. 

[9] Goral, Cindy M., Kenneth E. Torrance, Donald P. Greenberg, 
and Bennett Battaile. "Modeling the Interaction of Light 
Between Diffuse Surfaces," Proceedings of SIGGRAPH'84 
(Minneapolis, Minnesota, July 23-27, 1984), in Computer 
Graphics, 18(3), July 1984, pages 213-222. 

[10] Haralick, Robert M. "Automatic Remote Sensor Image Pro- 
cessing," Topics in Applied Physics, 11, 1976, pages 5-63. 

[11] Hartmann, Rudolf. Opera, William Morrow and Company, 
New York, N.Y., 1977. 

[12] Heckbert, Paul S. Fundamentals of Texture Mapping and 
Image Warping, Master's thesis. Department of EECS, UC 
Berkeley, Berkeley, CA 94720, June 1989. 

[13] Kaufman, John E. IES Lighting Handbook - Application 
Volume, Illuminating Engineering Society of North America, 
New York, 1987. 

[14] Max, Nelson L. "Computer Graphics Distortion for 
IMAX and OMNIMAX Projection," in Proceedings of Nico- 
graph'83, 1983. 

[15] Nishita, Tomoyuki, Yasuhiro Miyawaki, and Eihachiro Naka- 
mae. "A Shading Model for Atmospheric Scattering Con- 
sidering Luminous Intensity Distribution of Light Sources," 
Proceedings of SIGGRAPH'87 (Anaheim, California, July 
27-31. 1987), in Computer Graphics, 21(4), July 1987, 
pages 303-310. 

[16] Nishita, Tomoyuki and Eihachiro Nakamae. "Continu- 
ous Tone Representation of Three-Dimensional Objects Tak- 
ing Account of Shadows and Interreflections," Proceedings 
of SIGGRAPH'85 (San Francisco, California, July 22-26, 
1985), in Computer Graphics, 19(3), July 1985, pages 23-30, 

[17] Nishita, Tomoyuki, Isao Okamura, and Eihachiro Nakamae. 
"Shading Models for Point and Linear Sources," ACM Trans- 
actions on Graphics, 4(2), April 1985, pages 124-146. 

[ 1 8] Parker, W. Oren and Harvey K. Smith. Scene Design and Stage 
Lighting, Holt, Rinehart and Winston, Inc., New York, 1979. 



[19] Pilbrow, Richard. Stage Lighting, Von Nostrand Reinhold 
Company, New York, 1979. 

[20] Sellman, Hunton D. and Merrill Lessley. Essentials of Stage 
Lighting, Prentice-Hall, Inc, Englewood Cliffs, 1982. 

[21] Siegel, Robert and John R. Howell. Thermal Radiation Heat 
Transfer, Hemisphere Publishing Corp., Washington DC., 
1981. 

[22] Sillion, Francois and Claude Puech. "A General Two-Pass 
Method Integrating Specular and Diffuse Reflection," Pro- 
ceedings of SIGGRAPH'89 (Boston, Massachusetts, July 31 

- August 4, 1989), in Computer Graphics, 23(3), July 1989, 
pages 335-344. 

[23] Verbeck, Channing P. and Donald P. Greenberg. "A Compre- 
hensive Light-Source Description for Computer Graphics," 
IEEE Computer Graphics and Applications, 4(7), July 1984, 
pages 66-75. 

[24] Wallace, John R., Kelts A. Elmquist, and Eric A. Haines. 
"A Ray Tracing Algorithm for Progressive Radiosity," Pro- 
ceedings of SIGGRAPH*89 (Boston, Massachusetts, July 31 

- August 4, 1989), in Computer Graphics, 23(3), July 1989, 
pages 315-324. 

[25] Warn, David R. "Lighting Controls for Synthetic Images," 
Proceedings of SIGGRAPH '83 (Detroit, Michigan. July 25- 
29, 1983), in Computer Graphics, 17(3), July 1983, pages 13- 
21. 

[26] Watson, Lee. Lighting Design Handbook, McGraw-Hill, New 
York, 1990. 

[27] Wilfred, Thomas. Projected Scenery - A Technical Manual, 
The Drama Book Shop, New York, 1965. 

[28] Wolberg, George. Digital Image Warping, IEEE Computer 
Society Press., Los Alamitos, CA, 1990. 



Appendix A — Beam and Field Angles 

The beam angle is the central cone of light emitted from an instru- 
ment (Figure 14). The limit of the beam angle is usually defined as 




Figure 14: Relationship between the beam and field angles of a 
lighting instrument. 

that point where the light diminishes to 50 percent of its intensity 
when compared with the center of the beam. The field angle is de- 
scribed as that point where the light diminishes to 10 percent of the 
output of the center of the beam. 
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